Overview about this project

This system to manage Human genes data, three servers in JAPAN store and share this data together. When one of three has a problem, we still can recover data from two others.

1. Manage data and all divided part of it.
2. Allow user divide a file in a server to three path and send to others server (by sync action).
3. Upload a file to server
4. Download a file and merge it.

Download and upload big file (80Gb)

With the big file, if using JS as normal to upload and download, you will get out of memory. Solution is using old school approach/origin approach (jQuery/iframe/cookie based approach). Why? because in this way browser will handle the memory for us. Here I learn. https://stackoverflow.com/a/9834261/7185713

Combine 3 .tar file to 1 file only

Inside the 1 .tar file, we have hundered small file, cause system cannot send a big file(80Gb) to others server. They have to split it to a chunk. So when I combine 2 .tar file. Challenge I facing is:

1. I have to combine each small file or 1st .tar to each file of 2st .tar first. Then concat all of them in the order.
2. Disk space using is not more than 1 file. That mean, if I have two 80Gb file. Disk space I using to both save temporary file and combine file is not larger than 80Gb.
3. Run 3-5 thread to handle combine at same time and with out conflict in shared data of thread, keep it in order, and not out of memory.

=> All about algorithm and solution